{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在 Rental Listing Inquiries 数据上练习 xgboost 参数调优 数据说明： Rental Listing Inquiries 数据集是 Kaggle 平台上的一个分类竞赛任务，需要根据公寓的特征来预测其受欢迎程度（用户感兴趣程度分为高、中、低三类）。其中房屋的特征 x 共有 14 维，响应值 y 为用户对该公寓的感兴趣程度。评价标准为 logloss。 问题描述 采用xgboost模型完成商品分类（需进行参数调优）。 解题提示 为减轻大家对特征工程的入手难度，以及统一标准，数据请用课程网站提供的特征工程编码后的数据（RentListingInquries_FE_train.csv）或稀疏编码的形式（RentListingInquries_FE_train.bin）。xgboost既可以单独调用，也可以在sklearn框架下调用。大家可以随意选择。若采用xgboost单独调用使用方式，建议读取稀疏格式文件。 批改标准 独立调用xgboost或在sklearn框架下调用均可。\n",
    "模型训练：超参数调优 a) 初步确定弱学习器数目： 20分 b) 对树的最大深度（可选）和min_children_weight进行调优（可选）：20分 c) 对正则参数进行调优：20分 d) 重新调整弱学习器数目：10分 e) 行列重采样参数调整：10分\n",
    "调用模型进行测试10分\n",
    "生成测试结果文件10分"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 调整正则化参数：reg_alpha 和reg_lambda"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "导入模块"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from xgboost import XGBClassifier\n",
    "import xgboost as xgb\n",
    "\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "\n",
    "from sklearn.metrics import log_loss\n",
    "\n",
    "from matplotlib import pyplot\n",
    "import seaborn as sns\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#读取数据\n",
    "train = pd.read_csv(\"RentListingInquries_FE_train.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop ids and get labels\n",
    "y_train = train['interest_level']\n",
    "train = train.drop([\"interest_level\"], axis=1)\n",
    "X_train = np.array(train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prepare cross validation\n",
    "kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "第二轮参数调整得到的n_estimators最优值（250），其余参数继续默认值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'reg_alpha': [1.5, 2], 'reg_lambda': [0.5, 1, 2]}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#reg_alpha = [1e-3, 1e-2, 0.05, 0.1]    #default = 0\n",
    "#reg_lambda = [1e-3, 1e-2, 0.05, 0.1]   #default = 1\n",
    "\n",
    "reg_alpha = [ 1.5, 2]    #default = 0, 测试0.1,1，1.5，2\n",
    "reg_lambda = [0.5, 1, 2]      #default = 1，测试0.1， 0.5， 1，2\n",
    "\n",
    "param_test5_1 = dict(reg_alpha=reg_alpha, reg_lambda=reg_lambda)\n",
    "param_test5_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Anaconda3\\lib\\site-packages\\sklearn\\model_selection\\_search.py:761: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([mean: -0.58416, std: 0.00334, params: {'reg_alpha': 1.5, 'reg_lambda': 0.5},\n",
       "  mean: -0.58417, std: 0.00338, params: {'reg_alpha': 1.5, 'reg_lambda': 1},\n",
       "  mean: -0.58417, std: 0.00371, params: {'reg_alpha': 1.5, 'reg_lambda': 2},\n",
       "  mean: -0.58416, std: 0.00334, params: {'reg_alpha': 2, 'reg_lambda': 0.5},\n",
       "  mean: -0.58417, std: 0.00338, params: {'reg_alpha': 2, 'reg_lambda': 1},\n",
       "  mean: -0.58417, std: 0.00371, params: {'reg_alpha': 2, 'reg_lambda': 2}],\n",
       " {'reg_alpha': 1.5, 'reg_lambda': 0.5},\n",
       " -0.5841559743194963)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb5_1 = XGBClassifier(\n",
    "        learning_rate =0.1,\n",
    "        n_estimators=250,  #第二轮参数调整得到的n_estimators最优值\n",
    "        max_depth=5,\n",
    "        min_child_weight=1,\n",
    "        gamma=0,\n",
    "        subsample=0.7,\n",
    "        colsample_bytree=0.6,\n",
    "        colsample_bylevel = 0.7,\n",
    "        objective= 'multi:softprob',\n",
    "        seed=3)\n",
    "\n",
    "\n",
    "gsearch5_1 = GridSearchCV(xgb5_1, param_grid = param_test5_1, scoring='neg_log_loss',n_jobs=-1, cv=kfold)\n",
    "gsearch5_1.fit(X_train , y_train)\n",
    "\n",
    "gsearch5_1.grid_scores_, gsearch5_1.best_params_,     gsearch5_1.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([797.85700498, 777.78673306, 763.61248636, 786.33947268,\n",
       "        753.22281628, 654.46172609]),\n",
       " 'std_fit_time': array([ 10.09758564,  14.35703546,   2.03765806,  12.02590569,\n",
       "          4.35451259, 119.88518627]),\n",
       " 'mean_score_time': array([3.53077707, 1.43535089, 1.41076427, 1.4711904 , 1.41029687,\n",
       "        1.37036772]),\n",
       " 'std_score_time': array([1.53267301, 0.17183521, 0.19006806, 0.07951902, 0.07369521,\n",
       "        0.43743071]),\n",
       " 'param_reg_alpha': masked_array(data=[1.5, 1.5, 1.5, 2, 2, 2],\n",
       "              mask=[False, False, False, False, False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'param_reg_lambda': masked_array(data=[0.5, 1, 2, 0.5, 1, 2],\n",
       "              mask=[False, False, False, False, False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'params': [{'reg_alpha': 1.5, 'reg_lambda': 0.5},\n",
       "  {'reg_alpha': 1.5, 'reg_lambda': 1},\n",
       "  {'reg_alpha': 1.5, 'reg_lambda': 2},\n",
       "  {'reg_alpha': 2, 'reg_lambda': 0.5},\n",
       "  {'reg_alpha': 2, 'reg_lambda': 1},\n",
       "  {'reg_alpha': 2, 'reg_lambda': 2}],\n",
       " 'split0_test_score': array([-0.57828607, -0.57801683, -0.57729089, -0.57828607, -0.57801683,\n",
       "        -0.57729089]),\n",
       " 'split1_test_score': array([-0.58308886, -0.58321485, -0.58356289, -0.58308886, -0.58321485,\n",
       "        -0.58356289]),\n",
       " 'split2_test_score': array([-0.58478904, -0.58555004, -0.5854285 , -0.58478904, -0.58555004,\n",
       "        -0.5854285 ]),\n",
       " 'split3_test_score': array([-0.58730539, -0.58698725, -0.58721947, -0.58730539, -0.58698725,\n",
       "        -0.58721947]),\n",
       " 'split4_test_score': array([-0.58731147, -0.58706257, -0.58736941, -0.58731147, -0.58706257,\n",
       "        -0.58736941]),\n",
       " 'mean_test_score': array([-0.58415597, -0.58416613, -0.58417404, -0.58415597, -0.58416613,\n",
       "        -0.58417404]),\n",
       " 'std_test_score': array([0.0033428 , 0.00337589, 0.00370974, 0.0033428 , 0.00337589,\n",
       "        0.00370974]),\n",
       " 'rank_test_score': array([1, 3, 5, 1, 3, 5]),\n",
       " 'split0_train_score': array([-0.50125243, -0.50306946, -0.5064729 , -0.50125243, -0.50306946,\n",
       "        -0.5064729 ]),\n",
       " 'split1_train_score': array([-0.50077991, -0.50343587, -0.50632134, -0.50077991, -0.50343587,\n",
       "        -0.50632134]),\n",
       " 'split2_train_score': array([-0.50038213, -0.50201243, -0.5056325 , -0.50038213, -0.50201243,\n",
       "        -0.5056325 ]),\n",
       " 'split3_train_score': array([-0.50021942, -0.50249862, -0.50479077, -0.50021942, -0.50249862,\n",
       "        -0.50479077]),\n",
       " 'split4_train_score': array([-0.50064413, -0.50171496, -0.50545457, -0.50064413, -0.50171496,\n",
       "        -0.50545457]),\n",
       " 'mean_train_score': array([-0.5006556 , -0.50254627, -0.50573442, -0.5006556 , -0.50254627,\n",
       "        -0.50573442]),\n",
       " 'std_train_score': array([0.00035688, 0.00063929, 0.00061139, 0.00035688, 0.00063929,\n",
       "        0.00061139])}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gsearch5_1.cv_results_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best: -0.584156 using {'reg_alpha': 1.5, 'reg_lambda': 0.5}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAasAAAESCAYAAABD1PrQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd8lfX5//HXm7CHbNlIlD3CMEBQ61ecQFvRllawVcSBA7TaX6vVLkptrR1OQEQRqQtXBVQQ9YuTEkJYYUNkhqFsZJPk+v1xbvzGmAXk5D5JrufjkYfnfO7Pfd3XOZhc5/O5P+e+ZWY455xzsaxC2Ak455xzhfFi5ZxzLuZ5sXLOORfzvFg555yLeV6snHPOxTwvVs4552KeF6sSImmUpC2SFgc/A/Lpd4+k5ZKWSXpFUtVc25+UdCDH8wslLZSUKWlQMeX6kyCHbEmJxRHTOedOhxerKJB0kaTn89j0qJl1C35m5LFfM+AuINHMOgNxwOAc2xOBOrl22wTcALxcTOkDLAN+BHxajDGdc+6UebGKPRWBapIqAtWBrQCS4oB/APfm7GxmG8wsDcjOHUjSryXNl5Qm6U9FTcDMVprZ6tN5Ec45V5y8WJWskUHheE5S3dwbzWwL8E8io6VtwD4ze//EvsB0M9tWlANJuhxoA/QCugHnSrqwOF6Ec86VtIphJ1CWSJoHVAFqAvUkLQ423Qc8BfwZsOC//wJuzLV/XWAgEA/sBV6X9HNgNvAT4KKTSOfy4GdR8LwmkeL1qaQPgcZ57PNbM5t2EsdwzrkS4cWqGJlZb4icswJuMLMb8uon6RngnTw2XQqsN7MdQb//AOcBe4DWQLokgOqS0s2sdQHpCHjIzJ7OI89Li/qanHMuFvg0YAmR1CTH06uJLGLIbROQJKm6IlXpEmClmb1rZo3NrJWZtQIOFVKoAGYBN0qqGRy/maQzT/+VOOdcyfNiVXL+LmmppDSgL3APgKSmkmYAmNk84A1gIbCUyL/PhIKCSuopKYPINOHTkpYHsd4nskJwrqSlQdxaRUlU0tVBzD7Au5JmnfSrdc65YiS/RYhzzrlY5yMr55xzMc8XWBSTBg0aWKtWrcJOwznnSpUFCxbsNLOGhfXzYlVMWrVqRWpqathpOOdcqSJpY1H6+TSgc865mOfFyjnnXMzzYuWccy7m+TmrKDp+/DgZGRkcOXIk7FRcLlWrVqV58+ZUqlQp7FScc0XgxSqKMjIyqFWrFq1atSK4TJKLAWbGrl27yMjIID4+Pux0nHNF4NOAUXTkyBHq16/vhSrGSKJ+/fo+4nWuFPFiFWVeqGKT/7s4V7p4sXLOOXdKtm9OZ+7k35KdlRX1Y/k5K+ecc0Vm2dmsSH6Po3OeIuHA5zTE+CLtCtp0j+69XX1k5YrNxx9/zA9+8IPT7lOQVatW0adPH6pUqcI///nPfPvdcMMNxMfH061bN7p168bixYvz7eucK9zhg1+T8sYjrH+wO53eH8I5BxeQ2mQIXw6bF/VCBT6yKlfMDDOjQoXS+xmlXr16PPHEE0ydOrXQvv/4xz8YNGhQCWTlXNm1dcNqNr33OB22T6UXB1lXoRUpXUbRpd/NJNUo0l2HioUXqxLyp7eXs2Lr/mKN2bHpGfzxh50K7LNhwwb69+9P3759mTt3LnfffTfjx4/n6NGjnHPOOUyaNImaNWsyY8YMfvnLX9KgQQN69OjBunXreOedvG5mDCkpKdx9990cPnyYatWqMWnSJNq1a/etPqNGjeKLL75gy5YtbN68mXvvvZdbbrkFgAMHDjBo0CCWLVvGueeey4svvogkRo8ezdtvv83hw4c577zzePrpp7+zEOLMM8/kzDPP5N133z2Nd845VxDLzmb5nLc5Pnc8CQfnciYireYFVDn/djom9ePsED7wlt6P2K7IVq9ezfXXX88HH3zAxIkT+fDDD1m4cCGJiYk88sgjHDlyhFtvvZWZM2fy+eefs2PHjgLjtW/fnk8//ZRFixYxevRoHnjggTz7paWl8e677zJ37lxGjx7N1q1bAVi0aBGPPfYYK1asYN26dcyZMweAkSNHMn/+fJYtW8bhw4e/KZbjx49n/PjxJ/26f/vb35KQkMA999zD0aNHT3p/58qbg1/vZd5rf2fjgwl0/t/raXVoKSnNrmfnzfPp8eu36XTeABTSzIyPrEpIYSOgaDrrrLNISkrinXfeYcWKFZx//vkAHDt2jD59+rBq1SrOPvvsb74gO2TIECZMyP8Gxfv27WPo0KGsXbsWSRw/fjzPfgMHDqRatWpUq1aNvn37kpKSQp06dejVqxfNmzcHoFu3bmzYsIELLriAjz76iL///e8cOnSI3bt306lTJ374wx9y2223nfRrfuihh2jcuDHHjh1j+PDhPPzww/zhD3846TjOlQcZ6cvIeP8JOn71Nr05RHrcOaR0fZCEfjfSp1qNsNMDvFiVCzVqRP5nMzMuu+wyXnnllW9tX7Ro0UnF+/3vf0/fvn1566232LBhAxdddFGe/XJP4Z14XqVKlW/a4uLiyMzM5MiRI9xxxx2kpqbSokULRo0adVpf2m3SpMk3xxo2bFiBizGcK4+ys7JY9tlUbN7TdDmUQiMqkHbGhdT43gjaJV5C6xg7tx1b2bioSkpKYs6cOaSnpwNw6NAh1qxZQ/v27Vm3bh0bNmwA4NVXXy0wzr59+2jWrBkAzz//fL79pk2bxpEjR9i1axcff/wxPXv2zLfvicLUoEEDDhw4wBtvvHESr+y7tm3bBkQK9NSpU+ncufNpxXOurPh6326SX/krWx7sTMLHN9L88CpSWtzI3lsXcu7/m0r7XpeFNtVXEB9ZlSMNGzbk+eefZ8iQId+cw3nwwQdp27Yt48aNo1+/fjRo0IBevXoVGOfee+9l6NChPPLII1x88cX59uvVqxff//732bRpE7///e9p2rQpa9asybNvnTp1uOWWW+jSpQutWrX6VmE7cb7qtttuY/v27SQmJrJ//34qVKjwzbmvM844gwEDBvDss8/StGlTfvazn7Fjxw7MjG7dup3SOS/nypJNaxaz7f0n6LRjBkk6zJqKbUlN+BtdLh9KUtXqYadXKJlZ2DmUCYmJiZb7TsErV66kQ4cOIWV0cg4cOEDNmjUxM0aMGEGbNm245557TjneqFGjqFmzJr/61a+KMcviVZr+fZw7FdlZWSz95HWU8gwJR1I5ZnGk1b6YMy4aSdseF4WdHgCSFphZYmH9fGTlAHjmmWeYPHkyx44do3v37tx6661hp+ScO0X79uxk5YxxNE9/ia62nR3UZW7LW2kzYCSJjVuGnd4p8ZFVMSntI6u8TJo0iccff/xbbeeffz5jx44NKaPiVdr/fZzLbePKBWz/8Am67JxJdR1lVaWOHOh2IwmXXkflKlXDTi9PPrJyp23YsGEMGzYs7DSccwXIyswkbfYUKi14hs5HF9PYKpFW91Lq9h1J+64XhJ1esfFi5ZxzpdC+XV+ycsZYWq57he72FV9Sn7nxI2jXfwQ9z2wWdnrFLqrrEyX1k7RaUrqk3+SxvYqkV4Pt8yS1yrHt/qB9taQrCospKT6IsTaIWTlof1TS4uBnjaS9QXs3SXMlLZeUJumaHLGel7Q+x37dovMOOefcyVm3bB4pj/+Myk90JumLx9lbqRELez9G/d+uos/Qv1KvDBYqiOLISlIcMBa4DMgA5kuabmYrcnS7CdhjZq0lDQYeBq6R1BEYDHQCmgIfSmob7JNfzIeBR81siqTxQeynzOybJW2S7gS6B08PAdeb2VpJTYEFkmaZ2d5g+6/N7PS+7OOcc8Ug8/gx0j58iaqLJtLx2FKaWGWW1ruc+hffSccuSWGnVyKiOQ3YC0g3s3UAkqYAA4GcxWogMCp4/AYwRpHLHAwEppjZUWC9pPQgHnnFlLQSuBi4NugzOYj7VK6chgB/BDCzb77wY2ZbJX0FNAT24pxzMWDPjm2smvEk8etfpQc72UZDks/5BR0GjKBX/UZhp1eiojkN2AzYnON5RtCWZx8zywT2AfUL2De/9vrA3iBGnseSdBYQD8zOnaikXkBl4IsczX8JpgcflVQl9z7BfsMlpUpKLezir+VBSdzP6qWXXiIhIYGEhATOO+88lixZcsqxnItV6Us+Z/5jg6k+pgt91o9lZ5XmLDpvLGf+bhVJ142mdjkrVBDdkZXyaMu9Tj6/Pvm151VcC+qf02DgDTP71v2XJTUBXgCGmll20Hw/sJ1IAZsA3AeM/s4BzCYE20lMTIz57wCUhftZxcfH88knn1C3bl1mzpzJ8OHDmTdvXthpOXfajh87StoHL1Bj8UTaH1/BIavC4gbfp9Gld9K5Q6Eru8u8aBarDKBFjufNga359MmQVBGoDewuZN+82ncCdSRVDEZXeR1rMDAiZ4OkM4B3gd+ZWfKJdjPbFjw8KmkScPqXYZj5G9i+9LTDfEvjLtD/bwV2KWv3szrvvPO+eZyUlERGRsapvHPOxYyd2zezduYYztn4Gueymy1qRHKb/0eHAXfQu26DsNOLGdH8iD0faBOs0qtMpFhMz9VnOjA0eDwImG2RbylPBwYHqwXjgTZASn4xg30+CmIQxJx24iCS2gF1gbk52ioDbwH/NrPXcyYVjLYIzp9dBSw7rXciZGX1flYTJ06kf//+p/PWOBeaNQs/JvWRQZzxVDf6bBzP9qrxLLnwaZr8biVJP/sDtb1QfUvURlZmlilpJDALiAOeM7PlkkYDqWY2HZgIvBAsoNhNpPgQ9HuNyGKMTGDEiem7vGIGh7wPmCLpQWBREPuEIUQWbOScqvspcCFQX9INQdsNZrYYeElSQyLTi4uBk7+hUm6FjICiqSzez+qjjz5i4sSJfP7556f79jhXYo4eOcTS9ydzRtpztM1cw0GryqKGA2ly+V0ktPVvyBQkql8KNrMZwIxcbX/I8fgI8JN89v0L8JeixAza1/F/KwZzbxuVR9uLwIv59M//UuKlUFm7n1VaWho333wzM2fOpH79+ieVu3Nh2LF1A1/MeILWGW+SyF42qynJ7e6j04Db6F27XtjplQql90y7O2ll4X5WmzZt4kc/+hEvvPACbdu2zbOPc7HAsrNZlfIBC/51NXWe7kGvzc+xpVo70i56jma/W0bSkAeo5YWqyPxyS+VIWbif1ejRo9m1axd33HEHABUrViT3BYSdC9ORwwdJe+856i2bRPusL/jaqrGg0SCaX34XXVv7TUBPlV91vZiU9quu+/2snDs92zens37mE7Tf+hZ12c/GCi3Y3v56OvcfTo1adcJOL2b5VdfdSfH7WTl38iw7m5XzZnFkzjgSvv6cMzGW1OjDlj630en8H3JWKf5OY6zxkVUxKe0jq7z4/aycy9vhg1+z9L1nabD8ec7O3sA+arCy8VW0vOIumsa3Dzu9UsVHVjHCzL6zKq60KMv3s/IPae5UbN2wmk3vPU777dPoxQHWV2hFSuc/0qX/LSTVqBV2emWaF6soqlq1Krt27aJ+/fqltmCVRWbGrl27qFo1Nu+c6mKLZWez/L/vcPy/40k4+F/ORKTVvIAq599Ox6R+xPtUX4nwYhVFzZs3JyMjo9ArQriSV7Vq1W++mOxcXg4d2MfSGRNotOrfdM7exB5qkdLseuL730WPFq3DTq/c8WIVRZUqVfrmqhDOudJhy7rlbJ71BB2/nE5vDpEedw4pXf9MwhU30qd6zbDTK7e8WDnnyr3srCyWfz6NrOSnSTg0jzOpQNoZF1Ljgjto1/NSWvtUX+i8WDnnyq0D+/ewfMZ4mqx5gS7ZW9hFbVJa3MjZ/e/k3GY+KxJLvFg558qdzWuXsPX9J+j01bv01mHWVGxLare/0eXyoSRVrR52ei4PXqycc+VCdlYWSz95HaU8Q8KRVBpZHGm1+1LrwhG0SyxT164uk7xYOefKtP17d7Fixjiar32JrraNHdRlbstbaTNgJImNW4adnisiL1bOuTJp48oFbP/wCbrsnEmSjrKqUkdSu/0/Ei69jj5V/Dt2pY0XK+dcmZGVmUna7ClUWvAMnY8uprFVIq3OJdTpO5L23b4XdnruNHixcs6Vevt2fcnKmeNo+cXLdLev+JL6JLcaQdsBI+h5ZrOw03PFwIuVc67UWr98Hjv+90m67JpFko6xonIXtnd/gIRLf0ajSpXDTs8Vo6h+001SP0mrJaVL+k0e26tIejXYPk9Sqxzb7g/aV0u6orCYkuKDGGuDmJWD9kclLQ5+1kjam2OfoUH/tZKG5mg/V9LS4BhPyC/s51zMyDx+jIXvPc/yv15A/OuXk7DrPZbWu4wvfjyLjg98To/+w6joharMidrISlIcMBa4DMgA5kuabmYrcnS7CdhjZq0lDQYeBq6R1BEYDHQCmgIfSjpxD/P8Yj4MPGpmUySND2I/ZWbf3EFQ0p1A9+BxPeCPQCJgwIIg1h7gKWA4kAzMAPoBM4v5LXLOnYQ9O7axasYY4tdPoQc72UZDks++i/YDRtCrQeOw03NRFs1pwF5AupmtA5A0BRgI5CxWA4FRweM3gDHBKGYgMMXMjgLrJaUH8cgrpqSVwMXAtUGfyUHcp3LlNIRIgQK4AvjAzHYHsT4A+kn6GDjDzOYG7f8GrsKLlXOhSF8yh90fjaHrng/oo+Msq9KNbef+iYSLB9Okop/JKC+i+S/dDNic43kG0Du/PmaWKWkfUD9oT86174mzpHnFrA/sNbPMPPoDIOksIB6YXUB+zYKfjHyO7ZwrAcePHSXtgxeovngiHY6v4JBVYXGD79Po0jvp3KHQ+/S5MiiaxSqv8zy573iXX5/82vM6x1ZQ/5wGA2+YWdYpHvs7JA0nMl1Iy5b+5ULnTtfO7ZtZO3MM52x8jXPZzRY1IrnNL+nQ/w5612sYdnouRNEsVhlAixzPmwNb8+mTIakiUBvYXci+ebXvBOpIqhiMrvI61mBgRK5jX5Qr1sdBe/Nc7bljAWBmE4AJELmtfV59nHOFW7PwE/Z9Moaue2fTR5mkVT2XrT0fosv/DKKZT/U5olus5gNtJMUDW4gUi2tz9ZkODAXmAoOA2WZmkqYDL0t6hMgCizZACpFRz3diBvt8FMSYEsScduIgktoBdYPjnDAL+KukusHzy4H7zWy3pK8lJQHzgOuBJ4vlHXHOfePY0SOkzXqeWmnP0S5zNQetKosaDqTxZXeR0K5b2Om5GBO1YhWcgxpJpCjEAc+Z2XJJo4FUM5sOTAReCBZQ7CZSfAj6vUZkMUYmMOLE9F1eMYND3gdMkfQgsCiIfcIQIgs2vhn9BEXpz0SKKsDoE4stgNuB54FqRBZW+OIK54rJzq0bWTvzCdpsfoNE9rJZTUludy+dBtxO79r1wk7PxSjl+PvtTkNiYqKlpqaGnYZzMcmys1m9YDYHPh1H1/0fU0lZLKnWC/W6lc4XXk2FuLiwU3QhkbTAzApdNeOTwc65qDly+CBLZ02iztJJtM9K52urxoJGg2h++V10bd057PRcKeLFyjlX7L7M+IJ1M5+g/Zb/0JP9bKzQgnkdf0vn/sNJqlUn7PRcKeTFyjlXLCw7m5Up73Pk87EkfP05DTGW1OhDRtJtdL7gh5xVIapXd3NlnBcr59xpOXLoAGkzn6X+iufpmLWe/dQgtckQWl5xF93j24ednisjvFg5507Jto2r2fDeE3TYNpVeHGB9hVakdP4jXfrfQlKNWmGn58oYL1bOuSKz7GyW//cdjv93PAkH/8uZwJKaF1Dl/DvomNSPeJ/qc1Hixco5V6hDB/axdOYzNFo5mc7Zm9hDLVKaXUd8v7vo0bJN2Om5csCLlXMuX1vWrWTzrMfp+OV0enOQ9LhzSOn6ZxKuuJE+1WuGnZ4rR7xYOee+xbKzWfbZVLKSnybh0DwaIZbUupDq37uD9j0vo7VP9bkQeLFyzgFwYP8els98miar/02X7C3sojbzWgzjnP53cW6z+LDTc+WcFyvnyrnN6UvZMutxOn/1Dr11mDUV2zK/60MkXHEDfapWDzs95wAvVs6VS9lZWSz95E2UMoGEI/NpZHGk1e5LrQtH0C7x4rDTc+47vFg5V47s37uLFTOeovnaF+lq29hJHea2HE6bAXeS2NhvIOpilxcr58qBjasWsv3DJ+my412SdJRVFTuQ2u2XJFx2PX2qVA07PecK5cXKuTIqKzOTpR+9RlzqBLocXUQTq8iSOpdSp+9I2nf7XtjpOXdSvFg5V8bs272DlTPG0PKLV+hmX/IV9Zjb6nba9h9Jz0bNw07PuVPixcq5MmL98nl89b9jSNj1Hkk6xopKndnW/X4SLr2WPpWrhJ2ec6fFi5VzpVjm8WMsnf0KlRc8S6djaTSxSqTVu5z6F99Fxy5JYafnXLHxYuVcKbRnxzZWzRhD/PpX6c4OttOQ5LPvov2AEfRq0Djs9JwrdlG9boqkfpJWS0qX9Js8tleR9GqwfZ6kVjm23R+0r5Z0RWExJcUHMdYGMSvn2PZTSSskLZf0ctDWV9LiHD9HJF0VbHte0voc27pF5x1y7uR8kfZfUh4bQrUxXeizfgy7KjdlYZ8xNPjtCpKu/zN1vFC5MipqIytJccBY4DIgA5gvabqZrcjR7SZgj5m1ljQYeBi4RlJHYDDQCWgKfCipbbBPfjEfBh41symSxgexn5LUBrgfON/M9kg6E8DMPgK6BbnWA9KB93Pk9msze6O43xfnTtbxY0dJ+/BFqi+aSIfjy2liVVjSYABnXnInnTr2DDs950pENKcBewHpZrYOQNIUYCCQs1gNBEYFj98AxkhS0D7FzI4C6yWlB/HIK6aklcDFwLVBn8lB3KeAW4CxZrYHwMy+yiPXQcBMMzt0ui/aueKy68sM1sx4knM2vsa57GaLGpHc5pd06H8Hves1DDs950pUNItVM2BzjucZQO/8+phZpqR9QP2gPTnXvs2Cx3nFrA/sNbPMPPq3BZA0B4gDRpnZe7nyGAw8kqvtL5L+APwv8JugcH6LpOHAcICWLf3b/654rFn4Cfs+GUPXvbPpo0yWVunB1p5/pctFP6FZRT/N7MqnaP6frzzarIh98mvP6xxbQf0h8hrbABcBzYHPJHU2s70AkpoAXYBZOfa9H9gOVAYmAPcBo79zALMJwXYSExNzvzbniuzY0SOkvT+ZWkueo13mKg5aVRY1vJLGl/2CLu38lKlz0SxWGUCLHM+bA1vz6ZMhqSJQG9hdyL55te8E6kiqGIyucvbPAJLN7DiRKcXVRIrX/GD7T4G3gu0AmNm24OFRSZOAX53MC3euqHZu3cjamU/SZvPrJLKXzWpKcrt76TTgdnrXrhd2es7FjGgWq/lAG0nxwBYiU23X5uozHRgKzCVy3mi2mZmk6cDLkh4hssCiDZBCZAT1nZjBPh8FMaYEMacFx5gKDAGel9SAyLTguhw5DCEykvqGpCZmti04f3YVsOy03w3nApadzeoFsznw6TgS9n9MH2WxpFovtva6lc4XXk2LuLiwU3Qu5kStWAXnoEYSmV6LA54zs+WSRgOpZjYdmAi8ECyg2E2k+BD0e43IYoxMYISZZQHkFTM45H3AFEkPAouC2AR9L5e0AsgisspvVxCrFZGR2ie50n9JUkMixXExcFvxvTOuvDp65BBp702iztLnaJ+VztdWjYWNfkyzy++ia+suYafnXEyTWdFPtUiqANQ0s/3RS6l0SkxMtNTU1LDTcDHoy4wvWDfzSdpteZN67GdjhRZsb3cdnfoPp+YZdcNOz7lQSVpgZomF9St0ZBV8ifY2IqOSBUBtSY+Y2T9OP03nyibLzmZlyvsc+XwcCV9/RkOMJTX6sCXpNjpf8EPOqhDV7+M7V+YUZRqwo5ntl/QzYAaR6bYFgBcr53I5cugAaTOfpf6KyXTMWsd+apDa+BpaXPELup/dIez0nCu1ilKsKkmqRGShwRgzOy7Jl2k7l8P2TWtZP/MJOmz7D704wPoKZzGv0x/o0v9mkmrWDjs950q9ohSrp4ENwBLgU0lnAX7OypV7lp3N8rnvcvy/40k4MIeGwJKaF1DlvNvp2Kc/8T7V51yxKbRYmdkTwBM5mjZK6hu9lJyLbYcO7GPpzGc5c+VkOmdvZA+1SGl2HfH97qJHyzZhp+dcmVSUBRa/ACYBXwPPAt2B3/Dti746V+ZtWbeSzbMep+OX0+nNQb6IO5v5CaPp0u8m+lSvGXZ6zpVpRZkGvNHMHg9u09EQGEakeHmxcmWeZWez7PNpZCU/TcLBZBohltS6kOrfu4P2PS/jHJ/qc65EFKVYnbju3gBgkpktCa7s4FyZdWD/HpbPfJrGq1+gS3YGuzmDeS2GcU7/uzi3WXzY6TlX7hSlWC2Q9D4QD9wvqRaQHd20nAvP8jnv0uKDm+nNIdZWbMP8hL/S5Yob6FOtRtipOVduFaVY3UTkJoXrzOyQpPpEpgKdK3PSl8zhrPdvYldcA7YOeJF2Pfoin+pzLnRFWQ2YLak5cG0w+/eJmb0d9cycK2EZ6cuo+9YQDqgGVW+cxlnNzwk7JedcoNCPjJL+BvyCyEVlVwB3SXoo2ok5V5J2bt+EXvoRFcjm6JA3aeSFyrmYUpRpwAFANzPLBpA0mchVze8vcC/nSon9e3ex75kraZK9l4yBr9HWb3boXMwp6mR8nRyP/doxrsw4cvggm8cNpEXmJr64eDxte1wUdkrOuTwUZWT1ELAouLmhgAvxUZUrAzKPH2Plkz+h+7GlpPb8B4n/86OwU3LO5aMoCyxekfQx0JNIsbqPoo/InItJlp3NwnHD6HVoDsnt7iXpB8PDTsk5V4Ai3SnYzLYRuQU9AJI2AS2jlZRz0ZY88R767HmHuc1vpM+Q34adjnOuEKc6QvIrWLhSK/nlB+mz5Xnm1buSpBv/FXY6zrkiONViVaT7WUnqJ2m1pHRJv8ljexVJrwbb50lqlWPb/UH76uC6hAXGlBQfxFgbxKycY9tPJa2QtDy48/GJ9ixJi4Of6UWJ5Uq31LefJmnNP1hY43sk3jHJv/DrXCmR7zSgpCfJuyiJb68OzG//OGAscBmQAcyXNN3MVuTodhOwx8xaSxoMPAxcI6kjMBjoBDQFPpTUNtgnv5jVj65QAAAcSklEQVQPA4+a2RRJ44PYT0lqQ2RByPlmtkfSmTmOf9jM8lqnnGeswl6zi21pH71B19T7WV4lgY4jXyWuYpFmwZ1zMaCgj5WpRG5fn/snFbizCLF7Aelmts7MjgFTgIG5+gwEJgeP3wAuCS6SOxCYYmZHzWw9kB7EyzNmsM/FQQyCmFcFj28BxprZHgAz+6qgpAuJ5Uqp1amzaf3xHWyqeBYt7phKVb/On3OlSr4fLc1scn7biqgZsDnH8wygd359zCxT0j6gftCenGvfZsHjvGLWB/aaWWYe/dsCSJoDxAGjzOy9YFtVSalAJvA3M5taSKxvkTQcGA7QsqWvN4lVG1ctpNE717G7Ql1q3zKdM+rUDzsl59xJiuY8SF6LMHJPK+bXJ7/2vEaCBfWHyGtsA1wENAc+k9TZzPYCLc1sq6SzgdmSlgL7i5B3pNFsAjABIDExsUjn8VzJ2r45nSpTfkImFdF1U2nQuEXYKTnnTkE0zy5nADn/MjQHtubXR1JFIlfH2F3Avvm17wTqBDFyHysDmGZmx4MpxdVEihdmtjX47zrgYyJ3QS4olitF9u7cztFJV1HDDrLvR6/Q7OwOYafknDtF0SxW84E2wcq6ykQWTEzP1Wc6MDR4PAiYbWYWtA8OVgvGEykuKfnFDPb5KIhBEHNa8Hgq0BdAUgMi04LrJNWVVCVH+/nAikJiuVLi0IF9fDl+II2ztrPpiuc4J+G8sFNyzp2GQqcBJT2RR/M+INXM8v0jHpyDGgnMInKu6DkzWy5pdLDvdGAi8IKkdCIjqsHBvsslvUbkKu+ZwAgzywry+U7M4JD3AVMkPUjkQrsTg/ZZwOWSVgBZwK/NbJek84CnJWUTKdp/y7FSMb9YrhQ4fuwo6WN+TKfjq1ly3pP0OG9A2Ck5506TIgOJAjpIE4D2wOtB04+B5USm49aZ2d1RzbCUSExMtNTU1LDTKPeys7JY+PhPSdz/ISldRtHrx/eEnZJzrgCSFphZYmH9irLAojVw8YnVcZKeAt4n8l2npaeVpXPFyLKzSXn6dpL2f0hyqxEkeaFyrswoyjmrZkDOL6XUAJoG03JHo5KVc6cg+YXfk/TVqySf+VN6X/9g2Ok454pRUUZWfwcWB1deP3GLkL9KqgF8GMXcnCuylDcfo8/6MaSecSm9bh3vl1Fyrowpyi1CJkqaQeTqEQIeOLHkG/h1NJNzrigWvf8i56aNIq1aIgkjXqJCXFzYKTnnillRvxTcE/he8DgL/96RixEr5s6k45y7Sa/UltYj/0PlKlXDTsk5FwWFzpVI+hvwCyLLyFcAd0l6KNqJOVeYL5Ym03zWjWyPa0Sj26ZTvWbtsFNyzkVJUUZWA4BuZpYNIGkyke8e+a3tXWi2rFtJ7Tev4TDVqHLDVOo0aBx2Ss65KCrqWeictwTxj68uVDu3b8ZeuIqKZHLkmtdp3LJN2Ck556KsKCOrh4BFkj7i/1YD+qjKheLrfbvZ+8xAmmbvYdMPX6F9h3PDTsk5VwKKshrwlWDZek8ixeo+ontNQefydOTwQTaNvYq2mRtYcdHTdE28JOyUnHMlpEirAc1sGzkuQitpE+A3cHIlJiszkxVjBtPj2BJSz32YxL4/CTsl51wJOtURUl73j3IuKiw7m9SnbqTHwU9JbvsrEq+8LeyUnHMl7FSLld9o0JWYec/9it67pjG36VCSrv192Ok450KQ7zSgpCfJuyiJb68OdC5q5k15iKSMiaTU/T5JNz8WdjrOuZAUdM6qoPtd+L0wXNQtePdZeq58mEU1zqPHHc/79f6cK8fyLVZmNjl3m6TGZrY9uik5B0s/fYsuKfeyqnInOox8nYqVKoedknMuRCf7UXVGVLJwLoc1Cz/hnP+9lYy4FjS7YxpVq9cMOyXnXMhOtlj5KkAXVZvWLKbh9J+xt0Idzrh5OrXrNgg7JedcDDjZYvXMyXSW1E/Saknpkn6Tx/Yqkl4Nts+T1CrHtvuD9tWSrigspqT4IMbaIGblHNt+KmmFpOWSXg7aukmaG7SlSbomR//nJa2XtDj46XYyr9udmq+2rKfyy4PIpgLZP/sPDZqeFXZKzrkYcVLFyszGFbWvpDhgLNAf6AgMkdQxV7ebgD1m1hp4FHg42LcjMBjoBPQDxkmKKyTmw8CjZtYG2BPERlIbIpeHOt/MOgF3B/0PAdcHbf2AxyTlXOX4azPrFvwsLurrdqdm364vOTTxSmraAfZc/TLNW3cOOyXnXAyJ5vKqXkC6ma0zs2PAFGBgrj4DgRMLOd4ALpGkoH2KmR01s/VAehAvz5jBPhcHMQhiXhU8vgUYa2Z7AMzsq+C/a8xsbfB4K/AV0LBY3wFXJIcPfs228VfRNGsrGy5/htZdLwg7JedcjIlmsWoGbM7xPCNoy7OPmWUC+4D6BeybX3t9YG8QI/ex2gJtJc2RlCypX+5EJfUCKgNf5Gj+SzA9+KikKkV7ye5kHT92lDVjfkzbYytZ1udfdD7/h2Gn5JyLQdEsVnktxsj9JeP8+hRXO0SW57cBLgKGAM/mnO6T1AR4ARh24p5dRKYN2xO5eG89Ihfv/Q5JwyWlSkrdsWNHXl1cAbKzslg89jq6Hp7H/M6/o0e/G8JOyTkXo6JZrDKAFjmeNwe25tdHUkUi98raXcC++bXvBOoEMXIfKwOYZmbHgynF1USKF5LOAN4FfmdmySeCmtk2izgKTCIy/fgdZjbBzBLNLLFhQ59BPFkpz9xJz32zmHvWbfT+ya/CTsc5F8OiWazmA22CVXqViSyYmJ6rz3RgaPB4EDDbzCxoHxysFownUlxS8osZ7PNREIMg5rTg8VSgL4CkBkSmBdcF+78F/NvMXs+ZVDDaIjgXdhWw7LTfDfctyS/8gaTtLzGvwY9JGvpQ2Ok452JckW4RcirMLFPSSGAWEAc8Z2bLJY0GUs1sOjAReEFSOpER1eBg3+WSXgNWAJnACDPLAsgrZnDI+4Apkh4EFgWxCfpeLmkFkEVkld8uST8nciPJ+pJuCPreEKz8e0lSQyLTi4sBv8x3MUp560mSvnicBbUupuftz/hllJxzhVJkUOJOV2JioqWm+iUTC7P4w1fo/NkdrKzajXa/nEnlKlXDTsk5FyJJC8wssbB+/pHWlZiV82bR/rM7WVepNfEj/uOFyjlXZF6sXIlYv3wezWYOY0eFhjQYPpWaZ9QNOyXnXCnixcpF3dYNq6n5+jUcoQoVb5hKvTNzf93OOecK5sXKRdWuLzPImnwVlTnGoWtep8lZ7cJOyTlXCnmxclFzYP8edk8YSIPsnWwbMJlWHQo9h+qcc3nyYuWi4uiRQ2wYexXxmetY8z9P0r7XZWGn5JwrxbxYuWKXlZnJ8jGD6Xx0MYu6P0jXiweHnZJzrpTzYuWKlWVnkzr+Fnoc+ITk1vfQ86oRYafknCsDvFi5YpX8/H303vkf5jb5OUk/HxV2Os65MsKLlSs28177O302TWB+nf4k3fJk2Ok458oQL1auWCyYMYmey//K4mpJdB/xb7/en3OuWPlfFHfaln02jS7zfsXqyh1pN/INKlaqHHZKzrkyxouVOy1rF39G/IfD2RLXjKa3T6NajVphp+ScK4O8WLlTtjl9KfWnXst+1aLmzdOpXc9vQOmciw4vVu6U7Ni6gbiXfoSAzJ+9ScOmrcJOyTlXhnmxcidt356dHHh2ILWz97Nz4Eu0aNM17JScc2WcFyt3Uo4cOsDWcVfSLCuD9ZdOoE33C8NOyTlXDnixckWWefwYq8YMot2xFSzt/U86f29g2Ck558oJL1auSCw7m4Vjh9Lt0Fzmd3qAcwcMCzsl51w5EtViJamfpNWS0iX9Jo/tVSS9GmyfJ6lVjm33B+2rJV1RWExJ8UGMtUHMyjm2/VTSCknLJb2co31o0H+tpKE52s+VtDQ4xhOSVLzvTOmT/Mwv6LV3BnNb3ELvn94bdjrOuXImasVKUhwwFugPdASGSOqYq9tNwB4zaw08Cjwc7NsRGAx0AvoB4yTFFRLzYeBRM2sD7AliI6kNcD9wvpl1Au4O2usBfwR6A72AP0o6ca/1p4DhQJvgp19xvS+lUfJLf6LPtn8zr/5VJA37e9jpOOfKoWiOrHoB6Wa2zsyOAVOA3Cc5BgKTg8dvAJcEo5iBwBQzO2pm64H0IF6eMYN9Lg5iEMS8Knh8CzDWzPYAmNlXQfsVwAdmtjvY9gHQT1IT4Awzm2tmBvw7R6xyZ/60cSStfYSFNf+HxNsn+mWUnHOhiOZfnmbA5hzPM4K2PPuYWSawD6hfwL75tdcH9gYxch+rLdBW0hxJyZJOjJIKOkZGIXkDIGm4pFRJqTt27MirS6m2ZPYUui/8LcuqdKPTyCnEVawYdkrOuXIqmsUqr/M8VsQ+xdUOUJHIVN5FwBDgWUl1TjHWtxvNJphZopklNmxYtq7esCrlA9p+cifrK55NqxFTqVK1etgpOefKsWgWqwygRY7nzYGt+fWRVBGoDewuYN/82ncCdYIYuY+VAUwzs+PBlOJqIsWroGM0LyTvMm3DylSazBjKzgoNqDd8GjXPqFv4Ts45F0XRLFbzgTbBKr3KRBZMTM/VZzpwYhXeIGB2cJ5oOjA4WC0YT6S4pOQXM9jnoyAGQcxpweOpQF8ASQ2ITAuuA2YBl0uqGyysuByYZWbbgK8lJQXnwq7PEavM27ZxNdVf/QnHqEzc9W9Rv1Hzwndyzrkoi9pJCDPLlDSSSFGIA54zs+WSRgOpZjYdmAi8ICmdyIhqcLDvckmvASuATGCEmWUB5BUzOOR9wBRJDwKLgtjwf0VpBZAF/NrMdgWx/kykAAKMNrPdwePbgeeBasDM4KfM27NjG8cnX00NjrLrJ28RH98+7JSccw4ARQYl7nQlJiZaampq2GmcsoNf72XL45fR8vh61g94iQ69ryh8J+ecO02SFphZYmH9fB2y49jRI6wbczVnH09n1fee9ELlnIs5XqzKueysLJaOGUKXowtZ2O1PdLt0SNgpOefcd3ixKscsO5v5T93CuV/PJvnsu+h19V1hp+Scc3nyYlWOJU++n9473yS50RB6//xPYafjnHP58mJVTs17/V/02Tie+bUvp9fwsX4ZJedcTPO/UOXQolmTSVz2Z5ZU60W3ES9SIS4u7JScc65AXqzKmWVz3qbTf3/J2sodaDvyP1SqXCXslJxzrlBerMqR9CVzaPX+LWyNa0qT26ZSrUatsFNyzrki8WJVTmSkL6PuW0M4oJpUu3Eqtes3Cjsl55wrMi9W5cDOrRvRSz+iAtkcHfIGjZqfE3ZKzjl3UrxYlXH79+5i38SB1M3ey44rX+Ssdt3CTsk5506aF6sy7Mjhg2SMG0iLzE18ccnTtO1xUdgpOefcKfFiVUZlHj/Gyid/Qvujy0jr9TBdLrw67JScc+6UebEqgyw7m4XjbqD7oTmktL+XxO/fEnZKzjl3WrxYlUHJE++h1553mdv8RpKGPBB2Os45d9q8WJUxyS8/SJ8tzzOv3pUk3fivsNNxzrli4cWqDEmdPp6kNf9gYY3vkXjHJL/en3OuzPC/ZmVE2kdv0HXBAyyvnEDHka8SV7Fi2Ck551yxiWqxktRP0mpJ6ZJ+k8f2KpJeDbbPk9Qqx7b7g/bVkq4oLKak+CDG2iBm5aD9Bkk7JC0Ofm4O2vvmaFss6Yikq4Jtz0tan2NbTH85aXXqbFp/fAcbK7ai5YhpVK1WI+yUnHOuWEWtWEmKA8YC/YGOwBBJHXN1uwnYY2atgUeBh4N9OwKDgU5AP2CcpLhCYj4MPGpmbYA9QewTXjWzbsHPswBm9tGJNuBi4BDwfo59fp1jn8XF8qZEwcZVC2n0znXsrlCXOrdMo1btemGn5JxzxS6aI6teQLqZrTOzY8AUYGCuPgOBycHjN4BLJClon2JmR81sPZAexMszZrDPxUEMgphXnUSug4CZZnbopF9liLZvTqfqlEFkUhFdN5UGjVuEnZJzzkVFNItVM2BzjucZQVuefcwsE9gH1C9g3/za6wN7gxh5HevHktIkvSEpr7/og4FXcrX9JdjnUUl53kdD0nBJqZJSd+zYkVeXqNm7cztHJ11FdTvEvh+/SrOzO5To8Z1zriRFs1gpjzYrYp/iagd4G2hlZgnAh/zfSC6SgNQE6ALMytF8P9Ae6AnUA+7LIz5mNsHMEs0ssWHDhnl1iYpDB/bx5fgraZy1nc39JnFOl6QSO7ZzzoUhmsUqA8g5imkObM2vj6SKQG1gdwH75te+E6gTxPjWscxsl5kdDdqfAc7NlcNPgbfM7PiJBjPbZhFHgUlEph9jwvFjR0kf82NaH1/DivMfo2Of/mGn5JxzURfNYjUfaBOs0qtMZKpteq4+04GhweNBwGwzs6B9cLBaMB5oA6TkFzPY56MgBkHMafDNyOmEK4GVuXIYQq4pwBP7BOfCrgKWncLrL3bZWVksGXMtCUfmsyBhFN0v/3nYKTnnXImI2pdxzCxT0kgi02txwHNmtlzSaCDVzKYDE4EXJKUTGVENDvZdLuk1YAWQCYwwsyyAvGIGh7wPmCLpQWBREBvgLklXBnF2AzecyDFYKt8C+CRX+i9JakhkenExcFuxvCmnwbKzSXn6NpL2f8jc+BH0+fHdYafknHMlRpFBiTtdiYmJlpqaGrX4cyc/QJ/1Y0k+8xp63zber07hnCsTJC0ws8TC+vlfvFIg5c1H6bN+LKlnXEqvW5/yQuWcK3f8r16MW/T+i5yb9ifSqvYkYcRLVIiLCzsl55wrcV6sYtiKuTPpOOdu0iu1pfXIN6lcpWrYKTnnXCi8WMWoL5Ym0+K9YWyPa0yj26ZTvWbtsFNyzrnQeLGKQVvWraT2m9dwSNWpMmwqdRo0Djsl55wLlRerGLNz+2bshauoSCZHBr9B4xatw07JOedC58Uqhny9bzd7nxlIvew9bP/BvzmrfY+wU3LOuZjgxSpGHDl8kE1jr+KszA2kXzSO9omXhJ2Sc87FDC9WMSArM5MVY66h07ElLDn3ryT0HVT4Ts45V454sQqZZWeTOm4YPQ5+RnLbX5N4ZehXdnLOuZjjxSpk8577Fb13T2du06EkXfu7sNNxzrmY5MUqRJadjWUdJ6Xu90m6+bGw03HOuZgVtauuu8KpQgX63Pok2VlZfr0/55wrgP+FjAF+vT/nnCuYFyvnnHMxz4uVc865mOfFyjnnXMzzYuWccy7mebFyzjkX87xYOeeci3lerJxzzsU8mVnYOZQJknYAG8POI9AA2Bl2EoWI9RxjPT/wHItDrOcHsZ/j6eZ3lpk1LKyTF6sySFKqmSWGnUdBYj3HWM8PPMfiEOv5QeznWFL5+TSgc865mOfFyjnnXMzzYlU2TQg7gSKI9RxjPT/wHItDrOcHsZ9jieTn56ycc87FPB9ZOeeci3lerJxzzsU8L1almKR+klZLSpf0m3z6/FTSCknLJb0cazlKainpI0mLJKVJGlDC+T0n6StJy/LZLklPBPmnSeoRY/n9LMgrTdJ/JXUtyfyKkmOOfj0lZUkaVFK5BcctND9JF0laHPyefFKS+QXHL+zfubaktyUtCXIcVsL5tQh+T1cGx/9FHn2i+7tiZv5TCn+AOOAL4GygMrAE6JirTxtgEVA3eH5mDOY4Abg9eNwR2FDCOV4I9ACW5bN9ADATEJAEzIux/M7L8e/bv6TzK0qOOf5fmA3MAAbFUn5AHWAF0DJ4XqK/J0XM8QHg4eBxQ2A3ULkE82sC9Age1wLW5PG7HNXfFR9ZlV69gHQzW2dmx4ApwMBcfW4BxprZHgAz+yoGczTgjOBxbWBrCeaHmX1K5Bc/PwOBf1tEMlBHUpOSya7w/Mzsvyf+fYFkoHmJJPbtHAp7DwHuBN4ESvr/waLkdy3wHzPbFPSPxRwNqCVJQM2gb2ZJ5AZgZtvMbGHw+GtgJdAsV7eo/q54sSq9mgGbczzP4Lv/87QF2kqaIylZUr8Syy6iKDmOAn4uKYPIp+47Sya1IivKa4gVNxH5ZBtTJDUDrgbGh51LPtoCdSV9LGmBpOvDTigPY4AORD7MLQV+YWbZYSQiqRXQHZiXa1NUf1cqFlcgV+KUR1vu7yFUJDIVeBGRT9yfSepsZnujnNsJRclxCPC8mf1LUh/ghSDHUH4R81CU1xA6SX2JFKsLws4lD48B95lZVmRgEHMqAucClwDVgLmSks1sTbhpfcsVwGLgYuAc4ANJn5nZ/pJMQlJNIiPku/M4dlR/V7xYlV4ZQIscz5vz3Sm0DCDZzI4D6yWtJlK85pdMikXK8SagH4CZzZVUlciFMUt8KiYfRXkNoZKUADwL9DezXWHnk4dEYEpQqBoAAyRlmtnUcNP6Rgaw08wOAgclfQp0JXJeJlYMA/5mkZND6ZLWA+2BlJJKQFIlIoXqJTP7Tx5dovq74tOApdd8oI2keEmVgcHA9Fx9pgJ9ASQ1IDLdsS7GctxE5BMtkjoAVYEdJZhjYaYD1wcrnZKAfWa2LeykTpDUEvgPcF2MjQS+YWbxZtbKzFoBbwB3xFChApgGfE9SRUnVgd5EzsnEkpy/J42AdpTg73JwrmwisNLMHsmnW1R/V3xkVUqZWaakkcAsIiutnjOz5ZJGA6lmNj3YdrmkFUAW8OuS/ORdxBz/H/CMpHuITBncEHx6LBGSXiEyTdogOG/2R6BSkP94IufRBgDpwCEin3BLTBHy+wNQHxgXjFwyrYSv0F2EHENVWH5mtlLSe0AakA08a2YFLsMv6RyBPwPPS1pKZLrtPjMryduGnA9cByyVtDhoewBomSPHqP6u+OWWnHPOxTyfBnTOORfzvFg555yLeV6snHPOxTwvVs4552KeFyvnnHMxz4uVc865mOfFyrlyJLgVxjun28e5kubFyrkYEnz7338vncvFfymcC5mkVsFN7cYBC4HrJM2VtFDS68HFQ5E0QNIqSZ8HN7nLd/QjqZciN2NcFPy3XR59Rkl6QdJsSWsl3ZJjc01JbwTHeym43A6S/iBpvqRlkiacaHcu2rxYORcb2gH/Bi4jcnHfS82sB5AK/DK4wO/TRC5WewGRG/AVZBVwoZl1J3JJpr/m0y8B+D7QB/iDpKZBe3fgbiI3xDybyOV2AMaYWU8z60zkCuU/OOlX6twp8GsDOhcbNppZsqQfECkQc4JBS2VgLpErbK8zs/VB/1eA4QXEqw1MltSGyDUXK+XTb5qZHQYOS/qIyA0z9wIpZpYBEFwLrhXwOdBX0r1AdaAesBx4+9ResnNF58XKudhwMPivgA/MbEjOjZK6n2S8PwMfmdnVwc3yPs6nX+6Lg554fjRHWxZQMRjdjQMSzWyzpFFErpLvXNT5NKBzsSUZOF9SawBJ1SW1JTKtd3ZQeACuKSRObWBL8PiGAvoNlFRVUn0iV/0u6F5nJwrTzuA82qBCcnCu2Hixci6GmNkOIsXlFUlpRIpX+2Cq7g7gPUmfA18C+woI9XfgIUlziNyeJT8pwLvBcf5sZvneLC+4w/QzRG6rPpWSu4mnc36LEOdKC0k1zexAsAJvLLDWzB49jXijgANm9s/iytG5aPGRlXOlxy3BYoflRKb5ng45H+dKjI+snCvFJA0DfpGreY6ZjQgjH+eixYuVc865mOfTgM4552KeFyvnnHMxz4uVc865mOfFyjnnXMz7/7Jkx6fERgtyAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# summarize results\n",
    "print(\"Best: %f using %s\" % (gsearch5_1.best_score_, gsearch5_1.best_params_))\n",
    "test_means = gsearch5_1.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = gsearch5_1.cv_results_[ 'std_test_score' ]\n",
    "train_means = gsearch5_1.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = gsearch5_1.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "pd.DataFrame(gsearch5_1.cv_results_).to_csv('my_preds_reg_alpha_reg_lambda_1.csv')\n",
    "\n",
    "# plot results\n",
    "test_scores = np.array(test_means).reshape(len(reg_alpha), len(reg_lambda))\n",
    "train_scores = np.array(train_means).reshape(len(reg_alpha), len(reg_lambda))\n",
    "\n",
    "#log_reg_alpha = [0,0,0,0]\n",
    "#for index in range(len(reg_alpha)):\n",
    "#   log_reg_alpha[index] = math.log10(reg_alpha[index])\n",
    "    \n",
    "for i, value in enumerate(reg_alpha):\n",
    "    pyplot.plot(reg_lambda, -test_scores[i], label= 'reg_alpha:'   + str(value))\n",
    "#for i, value in enumerate(min_child_weight):\n",
    "#    pyplot.plot(max_depth, train_scores[i], label= 'train_min_child_weight:'   + str(value))\n",
    "    \n",
    "pyplot.legend()\n",
    "pyplot.xlabel( 'reg_alpha' )                                                                                                      \n",
    "pyplot.ylabel( '-Log Loss' )\n",
    "pyplot.savefig( 'reg_alpha_vs_reg_lambda1.png' )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
